Towards Unrestricted Lip Reading

Lip reading provides useful information in speech perception and language understanding, especially when the auditory speech is degraded. However, many current automatic lip reading systems impose some restrictions on users. In this paper, we present our research e orts, in the Interactive System Laboratory, towards unrestricted lip reading. We rst introduce a top-down approach to automatically track and extract lip regions. This technique makes it possible to acquire visual information in real-time without limiting user's freedom of movement. We then discuss normalization algorithms to preprocess images for di erent lightning conditions (global illumination and side illumination). We also compare di erent visual preprocessing methods such as raw image, Linear Discriminant Analysis (LDA), and Principle Component Analysis (PCA). We demonstrate the feasibility of the proposed methods by development of a modular system for exible human-computer interaction via both visual and acoustic speech. The system is based on an extension of an existing state-of-the-art speech recognition system, a modular Multiple State-Time Delayed Neural Network (MS-TDNN) system. We have developed adaptive combination methods at several di erent levels of the recognition network. The system can automatically track a speaker and extract his/her lip region in real-time. The system has been evaluated under di erent noisy conditions such as white noise, music, and mechanical noise. The experimental results indicate that the system can achieve up to 55% error reduction using additional visual information.